Software engineers often test new features and changes to their software applications before making the features and/or changes permanently available to application users. Testing helps engineers understand the effect of application changes on user traffic and the overall user experience.
When a software engineer tests an application change by running a live experiment using Internet or other network traffic, hundreds of metrics across several thousand different categories of traffic may be affected as illustrated in FIG. 1. For example, an experiment on an application could cause significant delays in user experience for users in a specific location, such as Japan. As shown in FIG. 1, an experiment may be run called “Mobile Instant Gratify” to decide if cache control affects the overall user experience. The application may be run once using cache control and another time without using cache control. Users in Japan may perform 50% fewer searches and have 50% fewer visits to the application when the application is run without using cache control as illustrated in FIG. 1 (101).
Customary systems can generate detailed statistics about application properties and user traffic to provide metrics about application performance. However, using customary techniques, analysts may need several hours to several days to explain one metric change. Analysts must create spreadsheets, aggregate data from multiple sources, and compute statistics by hand to determine if a particular change in traffic is significant to an experiment's analysis. Since there may be thousands of ways to categorize a user population and hundreds of metrics to measure, it is very difficult to determine the impact of a particular application change.